Issue1761
Created on 2011-06-15.19:58:08 by treaves, last changed 2018-11-04.16:21:03 by jeff.allen.
File name |
Uploaded |
Description |
Edit |
Remove |
leakdemo.py
|
jeff.allen,
2018-03-17.10:55:46
|
Timothy's code with a repetitive client program |
|
|
msg6553 (view) |
Author: Timothy Reaves (treaves) |
Date: 2011-06-15.19:58:07 |
|
Actually the version I am using is 2.5.2, but the Versions list does not offer that as a selection. I upgraded to 2.5.2 from 2.5.2rc4 to see if this issue is resolved.
This issue may be 1660 - that issue is marked closed, but, if it is not that issue, it is another, very similar one.
I use threading for work in my code. It's code copied from one of the Python doc pages.
class Worker(Thread):
def __init__(self, tasks):
Thread.__init__(self)
self.tasks = tasks
self.daemon = True
self.start()
def run(self):
while True:
func, args, kargs = self.tasks.get()
try:
func(*args, **kargs)
except Exception, exception:
logging.error(exception, extra={"sbs_app":"Worker"})
self.tasks.task_done()
class ThreadPool:
def __init__(self, num_threads):
self.tasks = Queue(num_threads)
for _ in range(num_threads):
Worker(self.tasks)
def add_task(self, func, *args, **kargs):
self.tasks.put((func, args, kargs), True)
def wait_completion(self):
self.tasks.join()
In this case, Jython is being executed by ODI, an Oracle product (I don't really think that's relevant, but...). I run this code with 32 threads, and then use the wait_for_completion(). All tasks run and finish fine. However, none of the 32 threads ever goes away.
I run jconsole, and when I kick off my process, I see the thread count increase, typically by 35 threads (some are ODI's). When the process has finished, and ODI finishes it's stuff, the delta is always exactly 32; it just keeps growing. Eventually, the VM runs out of memory, or the ulimit is hit.
In jconsole, all of the threads are still visible, and show:
Name: Thread
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57cac7
Total blocked: 38 Total waited: 140
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
org.python.modules._threading.Condition.Condition_wait(Condition.java:87)
org.python.modules._threading.Condition$Condition_wait_exposer.__call__(Unknown Source)
org.python.core.PyObject.__call__(PyObject.java:375)
Queue$py.get$12(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/Queue.py:179)
Queue$py.call_function(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/Queue.py)
org.python.core.PyTableCode.call(PyTableCode.java:165)
org.python.core.PyBaseCode.call(PyBaseCode.java:301)
org.python.core.PyBaseCode.call(PyBaseCode.java:127)
org.python.core.PyFunction.__call__(PyFunction.java:317)
org.python.core.PyMethod.__call__(PyMethod.java:109)
sbs.threadsupport$py.run$3(/opt/csw/lib/python/site-packages/ImageProcessingPipeline-2.3.1-py2.6.egg/sbs/threadsupport.py:44)
sbs.threadsupport$py.call_function(/opt/csw/lib/python/site-packages/ImageProcessingPipeline-2.3.1-py2.6.egg/sbs/threadsupport.py)
org.python.core.PyTableCode.call(PyTableCode.java:165)
org.python.core.PyBaseCode.call(PyBaseCode.java:134)
org.python.core.PyFunction.__call__(PyFunction.java:317)
org.python.core.PyMethod.__call__(PyMethod.java:109)
threading$py._Thread__bootstrap$33(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/threading.py:223)
threading$py.call_function(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/threading.py)
org.python.core.PyTableCode.call(PyTableCode.java:165)
org.python.core.PyBaseCode.call(PyBaseCode.java:301)
org.python.core.PyBaseCode.call(PyBaseCode.java:194)
org.python.core.PyFunction.__call__(PyFunction.java:387)
org.python.core.PyMethod.instancemethod___call__(PyMethod.java:220)
org.python.core.PyMethod.__call__(PyMethod.java:211)
org.python.core.PyMethod.__call__(PyMethod.java:201)
org.python.core.PyMethod.__call__(PyMethod.java:196)
org.python.core.FunctionThread.run(FunctionThread.java:21)
Line 223 in threading.py is the except clause of the attempt at deleting self, when an exception has occurred. So for some reason, the threads do not seem able to self-destruct.
Any help getting this taken care will be greatly appreciated.
|
msg7632 (view) |
Author: Alan Kennedy (amak) |
Date: 2013-02-06.21:38:16 |
|
Now that this issue
http://bugs.jython.org/issue1761
Has been fixed, is this one still an issue?
|
msg7635 (view) |
Author: Timothy Reaves (treaves) |
Date: 2013-02-07.13:15:53 |
|
What makes you think this issue has been fixed?
|
msg7636 (view) |
Author: Alan Kennedy (amak) |
Date: 2013-02-07.13:30:49 |
|
Sorry, bad paste of URL.
Now that issue #1660 has been fixed
http://bugs.jython.org/issue1660
Is this issue (#1761) still happening?
|
msg7637 (view) |
Author: Timothy Reaves (treaves) |
Date: 2013-02-07.14:42:40 |
|
I see. :)
Yes, this issue still happens with 2.5.2. 1660 was closed in 2010, yet this bug still clearly existed in 2011 when I commented on 1660. I wasn't sure if this defect was a new one (a regression introduced after 1660 was fixed), or if 1660 was never actually fixed (the fix was defective itself).
But this thread consumption still occurs, and with JConsole is easy to see. I've had to write my own thread code to get around this.
|
msg7638 (view) |
Author: Frank Wierzbicki (fwierzbicki) |
Date: 2013-02-07.18:01:06 |
|
#1660 was *just* fixed, hopefully it resolves this issue as well (but no binary release has this fix yet). I am in the process of finding any more low hanging fixes - as soon as I am done with that I will be putting out a 2.5.4 rc and a 2.7 beta.
|
msg7639 (view) |
Author: Frank Wierzbicki (fwierzbicki) |
Date: 2013-02-07.18:05:13 |
|
oops, I forgot, #1660 is only for 2.7 as it is an API change and is not backwards compatible, so testing for this should be on the 2.7 beta.
|
msg11821 (view) |
Author: Jeff Allen (jeff.allen) |
Date: 2018-03-17.10:55:46 |
|
I checked this on 2.7.2a1, observed with jvisualvm, and I can reproduce what I understand is the OP's issue. "When the process has finished, and ODI finishes it's stuff, the delta is always exactly 32; it just keeps growing." So the OP's application is run repeatedly in the same JVM, and each time the threads remain alive (parked).
My test program is attached. It creates successive pools of 32 threads. A given pool behaves in the textbook way, but when the pool is dereferenced the JVM threads of the old one persist, so the JVM thread count shows a rising staircase. Other garbage collection is clearly going on happily.
My difficulty is to know whether that is actually incorrect behaviour. Worker.run does not return, so the thread should not finish. This stackoverflow post, with almost identical code (run on CPython I assume), is helpful:
https://stackoverflow.com/questions/18610144/python-threads-are-not-being-garbage-collected
I'm taking off 2.5 as a target as we won't fix that as far as I know.
|
msg12172 (view) |
Author: Jeff Allen (jeff.allen) |
Date: 2018-11-04.16:21:03 |
|
This is really old, and I think based in a misunderstanding identified above.
|
|
Date |
User |
Action |
Args |
2018-11-04 16:21:03 | jeff.allen | set | status: pending -> closed messages:
+ msg12172 |
2018-03-17 10:58:46 | jeff.allen | set | files:
- unnamed |
2018-03-17 10:58:36 | jeff.allen | set | resolution: fixed -> invalid |
2018-03-17 10:55:47 | jeff.allen | set | status: open -> pending files:
+ leakdemo.py versions:
+ Jython 2.7, - Jython 2.5 nosy:
+ jeff.allen messages:
+ msg11821 resolution: remind -> fixed |
2014-12-27 17:24:20 | zyasoft | set | nosy:
+ zyasoft |
2014-09-26 05:35:21 | zyasoft | set | assignee: fwierzbicki -> resolution: remind |
2013-02-20 00:07:39 | fwierzbicki | set | versions:
+ Jython 2.5, - 2.5.2rc |
2013-02-07 18:05:13 | fwierzbicki | set | messages:
+ msg7639 |
2013-02-07 18:01:44 | fwierzbicki | set | priority: normal assignee: fwierzbicki |
2013-02-07 18:01:06 | fwierzbicki | set | nosy:
+ fwierzbicki messages:
+ msg7638 |
2013-02-07 14:42:40 | treaves | set | messages:
+ msg7637 |
2013-02-07 13:30:49 | amak | set | files:
+ unnamed messages:
+ msg7636 |
2013-02-07 13:15:53 | treaves | set | messages:
+ msg7635 |
2013-02-06 21:38:16 | amak | set | nosy:
+ amak messages:
+ msg7632 |
2011-08-11 13:56:23 | crankycoder | set | nosy:
+ crankycoder |
2011-06-16 20:54:18 | treaves | set | type: crash versions:
+ 2.5.2rc, - Deferred |
2011-06-15 19:58:08 | treaves | create | |
|